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Fig. 6 
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Fig. 11 
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Fig. 30 
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Fig. 34 
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DYNAMICALLY RELOCATABLE TILEABLE 
DISPLAYS 

FIELD OF THE INVENTION 

TTie present invention relates to a dynamically relocatable 
tileable display apparatus. More particularly, the present 
invention relates to autonomous tileable computer displays 
capable of supporting optical, radio, or mechanical inter- 
connections. 

BACKGROUND AND SUMMARY OF THE 
INVENTION 

Reliably, quickly, and intuitively transmitting complex 
commands to small portable computers can be difficult. 
Small computer devices do not generally have sufficient 
computer processing power to respond reliably to voice or 
handwritten (pen based) commands, keyboards are often 
absent or too small for accurate finger input, and conven- 
tional buttons are too large or support too limited a com- 
mand instruction set. User interface techniques that rely on 
bulky external modules (full size infrared linked keyboards, 
tethered data gloves, or camera based gestural recognition 
equipment, for example) are expensive, often not readily 
available outside selected sites, and probably too awkward 
for widespread use in conjunction with consumer level 
portable computing devices. 

User interface designers for portable computers have 
attempted to compensate for some of these problems by 
constructing devices that rely on various spatial, positional, 
or environmental cues that manually or automatically allow 
for activation of various modes in the device. For example, 
some laptop computers use the action of opening/closing the 
lid to initiate automatic bootup/powerdown of the computer 
without requiring any additional signal input (eg, such as 35 
depressing a "start" button or typing "_o_g_o_n'' on a 
keyboard) from a user. Alternatively, the use of small 
portable computers that automatically switch control modes 
depending on position, orientation, or room location have 
been investigated. Buttonless manual control of a portable 40 
computer through deliberate user actions such as tilting the 
portable computer have also been described. 

However, all these solutions for interfacing with small 
portable computers have generally been limited in scope and 
functionality. What is needed is a user interface system 45 
suitable for even very small portable computers (having 
volumetric dimensions on the order of a one cubic 
centimeter) that is powerful, can be intuitively operated by 
an ordinary user with little training, and is still readily 
capable of modification or extension by the user. The present 50 
invention meets these requirements by providing a to 
manipulatory user interface that responds to a user twisting, 
folding, bending, squeezing, shaking, tilting, spinning, 
lifting, or otherwise physically manipulating the computer. 

In the manipulatory user interface system of the present 55 
invention, the most basic level of manipulation is known as 
a "senseme" A senseme is defined as a single indivisible 
type of physical manipulation. A partial list of categories of 
sensemes include material transformations such as 
squeezing, twisting, stretching; local spatial transformations eu 
such as translation, rotation, orbiting; and environmental 
transformations based on temperature, light level, or vibra- 
tion. For example, a small portable computer may support a 
deformable piece having multiple embedded sensors that 
detect folding, twisting, or bending of the deformable piece 6S 
by a user. This computer can also contain a number of 
acce Urometers that sense relative spatial information; 
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gyroscopic, radio or infrared positional sensors for deter- 
mining absolute position; and various thermal or photosen- 
sors that respectively detect temperature and light level 
changes. Intentional or unintentional modifications detected 
s by one or more of these sensor systems can provide the basis 
for a powerful user interface scheme. 

As will be appreciated, each senseme category contains 
many individually distinguishable members. For example, 
consider the category of senseme known as a "pinch", a 
Q structural transformation generally completed by a user 
squeezing the deformable piece between a forefinger and 
thumb. A pinch can be modified by varying its speed (quick 
or slow pinch), magnitude/intensity (light or hard pinch), 
portion of deformable piece pinched (top, bottom, or center 
s of deformable piece pinched), or even portion of body used 
to pinch (right handed pinch or left handed pinch), with each 
modification being distinguishable as a senseme capable of 
being mapped onto a computer control command. 

Although the wide variety of easily distinguishable sen- 
, 0 semes would alone provide a powerful user interface to a 
computer, the present invention further extends the flexibil- 
ity of the senseme based user interface by supporting 
computer control based on a "morpheme" input. The mor- 
pheme is a temporally synchronous (or overlapping 
£5 asynchronous) tuple of one or more sensemes. Note that a 
morpheme can (and often will) contain more than one 
senseme. The sensemes combined into a morpheme can 
come either from the same category (the user pinches with 
a right hand while tapping with a left hand finger), or 
J0 different categories (the user pinches the deformable piece 
with a right hand while modifying the spatial position of the 
portable computer by tilting it forward). 

Any morpheme can in turn be extended by participation 
in a "sentence". A sentence is defined as a sequence of one 
or more temporally disjoint morphemes. The sentence level 
allows definition of a physical manipulatory grammar by 
appropriate choice of morpheme sequence, and corollary 
rules governing, for example, use of active (verb like) 
morphemes, naming (noun) morphemes, or cuoneciors. 
Other possible grammar constructs used in sentences may 
include those based on "home" systems. Home systems are 
general-purpose gestural languages, whose grammar and 
syntax are not borrowed in any way from a host language. 
Examples of these languages are gestural languages devel- 
oped by deaf children of hearing parents who have not been 
exposed to American Sign Language (ASL), and the "plains 
talk" of North American Indians, which was used as a trade 
language. 

Accordingly, the present invention provides a method for 
inputting information to a computer connected to a deform- 
able piece that can be manipulated, and optionally to various 
position sensors (both relative and absolute), pressure 
sensors, thermal sensors, or even light sensors. The method 
comprises the steps of manipulating the deformable piece to 
provide a first morpheme input to the computer, with the first 
morpheme input normally triggering a first default action by 
the computer. The deformable piece may also be manipu- 
lated to provide a second morpheme input to the computer, 
with the second morpheme input converting the normally 
triggered first default action to a second action. The first and 
second morphemes (and any subsequent morphemes) 
together form a sentence that urn be interpreted as a com- 
mand to implement a computer controlled action, whether it 
be to unlock an electronically controlled door, display a 
graphical image on a computer display, or begin logging on 
to a computer network. Advantageously, such a user inter- 
face system is well suited for interaction with small com- 
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puter devices, and may even be cross-cultural to a limited FIGS. 3-16 schematically illustrate various preferred 
extent, with ability to squeeze or pinch being universal classes of physical manipulation morphemes; 
human attributes. FIG. 17 is a graphical diagram illustrating various spatial 
The present invention is particularly useful for portable morphemes, with a first axis illustrating complexity of 
computers that can be held in the palm of a hand. If the 5 sense me tuples required to form a morpheme, and a second 
portable computer is partially or completely surrounded with axis i]i us i rat jng the degree of positional information needed 
a deformable material having embedded or contact pressure/ {Q support lhe physical manipulation (moving along a con- 
deformation sensors, a user is able to gesture with or tjnuum from reUtive IocftI measuremenl along a s j ng i e 
manipulate the whole computer in order to achieve some ^ dimensk)n |Q absolule ^ obal measurement with six 
desired result. Material deformation can be implemented a 1Q H freedom determined) 
various scales. For instance, a computer and connected fe a + _ 
deformation sensors mounted inside a flexible tube could FIGS. 18-26 schematically illustrate various preferred 
respond to right angle bending of the lube, or even complex spatial manipulation morphemes; 

knotting or looping of the tube. In practice however, only FIG. 27 is a graphical illustration showing increasing 

minute surface deformations are required, being just suffi- degrees of sensor system complexity that can be used to 

cient to provide tactile feedback to pressing, pinching, or 1 detect various categories of environmental stimuli, including 

bending type manipulations. In either case, however, the ii gnt effects, thermal effects, the electromagnetic 

measurement of location and pressure applied to a surface is environment, and the vibratory/acoustic environment; 

^ inUraCti0D FIG.28isa^ 

Sml^^ ~ ? les 7 quir ' d 10 ioL a raorpheme ' and iDcreasing 

a computer, a feedback module to provide visual, auditory, level of P h y acal 

or tactile feedback to a user (e.g., processor connected LCD FIGS. 29-34 schematically illustrate various preferred 

display, audio speaker, or tactile display to present Braille or 25 manipulation morphemes for multiple interacting devices; 

other conventional touch interface), and co- mounted FIGS. 35 and 36 are schematic diagrams illustrating 

graspable, deformable piece partially or completely sur- "squeeze" and "tilt" morphemes applicable to a portable 

rounding the feedback module. In addition, various thermal computer; 

or pressure sensors are mounted to detect handedness of a piG. 37 is a schematic diagram illustrating tilt and 

user, with the grasping hand generally being the non- 30 squcc2c morphemes used to control view of large two 

dominant hand for the user. Depending on the handedness ol ti ; mensiona i data scls with a relatively small display of a 

the user (which can be considered as a morpheme), the p tir t a blc computer; 

displayed data structure is modified. For example, lexi ' . nf *. Mo 

,7 F , J , T ^ . . . ,- n FIG. 38 is a schematic illustration representing a portable 

displayed on an LCD disp ay screen may be automatically K . .■ <■■ 

L f ! . . , . n u nt .*A nn rJ , C computer having a disp ay ready to receive annotations trom 

shifted nghtward on the screen to allow pen based anno la- 35 • h h h h 

lion on the left side of the screen, thereby aiding left handed a n S nt nanded user i 

users FIG. 39 is a schematic illustration representing a portable 

Physically manipulatable user interfaces additionally pro- computer having a display ready to receive annotations from 

vide an opportunity for multiple devices to interact in a user a lcft nandcd uscr J 

friendly manner. For example, a tileable display system 40 FIG. 40 is an electronic schematic illustrating components 

having multiple displays with touch sensitive deformable of the pressure and tilt sensitive modules of a portable 

pieces surrounding the display can be used to organize computer such as illustrated in FIGS. 35-39. 

documents based on relative position of contact of each FIGS. 41 and 42 are schematic illustrations of a scanner/ 

display. For example, if two displays initially showing printer/copier using a paper shaped display interface to 

separate data structures (e,g two different pages from two 45 support morphemic input; 

different electronic books) are brought together in side by FIGS 43.45 m schematic illustrations of tilable displays 

side touching contact, the displayed visual information can capa ble of supporting morphemic input; 

alter (e.g. by displaying adjacent pages of a single electronic nG ^ iUustrates tical and ttems suitable for 

book). As will be appreciated, one can also use multiple ^ . n conjunction with ti i eab i e displays such as illustrated 

computers connected to physically manipulatable control 50 ^ fa~4S' 

elements to construct complex commands for organizing " 

* p FIG. 47 illustrates radio transponders suitable tor use in 

data structures. . conjunction with tileable displays such as illustrated in 

Additional functions, objects, advantages, and leatures ot 43,45. and 

the present invention will become apparent from consider- _ '/ . 

ation of the following description and drawings of preferred 55 FIG. 48 illustrates addressing ot multiple tileable dis- 
embodiments. 



plays. 



BRIEF DESCRIPTION OF THE DRAWINGS DETAILED DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic diagram of an approximately FIG. 1 illustrates an embodiment of the present invention 
spherical hand holdable portable computer having a deform- eo suitable for supporting a morphemic user interface grammar, 
able surface, a status display, and a pressure sensor array for Support of the grammar can require detection of a users 
detecting surface deformations; physical manipulation of a device, detection of relative or 

FIG. 2 is a graphical diagram illustrating various physical absolute spatial location of the device, detection of various 
manipulation morphemes, with axes respectively illustrating environmental factors acting on the device, and even detec- 
complexity of senseme tuples required to form a morpheme, 65 don and interaction with multiple devices or external com- 
and plasticity of a device required to support the physical puter networks. As illustrated, a device 10 has a deformable 
manipulation; surface 20 with an underlying deformation sensor mesh 22 
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far detecting surface deformation across or within multiple twisting, folding, curling, or crumpling actions, and 

subregions of the deforroable surface 20. The deformation have been described in conjunction with U.S. Pat. No. 

sensor mesh 22 is connected to an internally contained 5,389,945, assigned to Xerox Corp., the disclosure of 

processor 24 having associated memory system 26. For which is herein specifically incorporated by reference, 

detecting various positional or environmental variables, a 5 The deformation sensor mesh 22 can be embedded within, 

sensing system 28 is also provided. The illustrated device or positioned to contact the deformable surface 20. The 

further includes a feedback module 33, which may include deformation sensor mesh 22 can include an array of mdi- 

ii • *ui ia «r o nnn „;«,«t vidual congressional or tensional strain sensors, or 

an externally vxsiblc status d * 30 alternatively, embedded or attached positional sensors. For 

feedback module 31 (typically delivering auditory or tactile ap pf ica tions, continuous sensors (e.g. bilayer sheets 

feedback). In the illustrated device, a communications sys- 10 of J£ m seDSOrs) may be employed. One particularly 

tem 32 for reception or transmission of information to other ugeful continuous sensor ty p e uses multiple capacitance or 

electronic or computing devices is also provided. All these resistance strips, with deformation pressure resulting in a 

components can be powered by a power supply 25, which is positionally localizable analog signal proportional to the 

usually an internally mounted rechargeable battery of con- applied deformation pressure. Various sensor types can be 

ventional construction. 15 used, including simple capacitance sensors, resistive strain 

Although the device 10 is illustrated as having an approxi- sensors, analog or digital pressure switches, inductive 

mately spheroidal and unitary mass, various other shapes are sensors, or even fluid flow sensors. Depending on the sensor 

contemplated to be within the scope of the present invention. type employed, sensor data can be directly fed to the 

For example, the overall shape may be similar to various processor 24 in digital form, or be transformed to digital 

rectangular prisms, or can be ellipsoidal, toroidal, planar, or 20 format by an general purpose analog/digital converter that 

even be malleable enough to support a wide range of user typically provides a 4 or 8 bit range (although as few as one 

defined irregular shapes. In addition, multiple cooperating or as many as 32 bits may be required by various 

shape elements are contemplated using conventional designs applications). An analog to digital converter may be internal 

that permit interlocking of multiple shape elements (e.g to the processor 24 or provided as an external module. As 

using a ball and socket, a lock and key, or slidable or ^ ^ ° e appreciated, the sensor mesh 22 is intended to 

rotatable interlocked components). include combinations of sensors and sensor types, which can 

4 . , - , • in t - r t .„ be used over the whole or part of the deformable surface 20. 

Whatever the shape , of dev.ee 10, tor operation o the ona| q{ environ ^ enlal sensor s lem a can also 

present invention the device 10 is completely or parttally be J device w ^ modes can ^ 

enveloped by the deformable surface 20. The present mven- jq ^ ^ abso j ute or relative positional informa . 

Uon supports use of a great vanety of designs and materials J determined g gyroscopic sensorS) accelerometers, or 

for the deformab e surface 20, depending on the required infrared ranging techniques. Environmental 

plasticity, durability, longevity, and of course, cost con- . . ^ r t r 7. . nna , , 

F t> * , * sensors, mcluding conventional light, image, thermal, 

straints For example contemplated designs for deformable ^ * or sensors can ^ be 

surface 20 include, but are not limited to: 35 ^ D % ending on the desire d application, even costly 

a closed or open celled polymeric foam material having a environm ental or positional sensors such as those incorpo- 

wall thickness of millimeters to centimeters, with thin- rat - differential GPS positioning, image analysis or 

ner walled embodiments being supported (e.g. by adhe- recognition, acoustic or voice identification, or differential 

sive attachment) by an internal hard shell (constructed thermal cm ^ ^ ag morphemic mput . Such 

from polymeric or metallic materials), and those ^ morphemic inpul> taken in conjunction with morphemic 

thicker walled embodiments directly supporting (by, inpul ag detected by sensor mesn 22> can enhance precision 

e.g. brackets or supports) internal components such as and flexi5ility of a users control of device 10. 

processor 24. Suitable foams may include those com- M ilhjslraled> bolh system 28 and sensor mesh 22 

posed in whole or in part of widely available synthetic are connecled to the processor 2 4 and associated memory 

rubbers such as polychloroprene (neoprene), AS 2 6. The processor 24 and memory 26 are typicaUy mounted 

polystyrenes, rubber or nitrite rubber Uiicx loams, within the deformable surface 20, by either direct attachment 

polysiloxanes, block polymers including siyrene- [Q lhe detbrmable surface 2 () or by attachment to a hard 

butadiene or styrene isoprene, or any oilier conven- casing posilioned wilhin lhe deformable surface 20. Con- 

tional material having good elaslicily and dekirmabil- vem ional CISC or RISC processors can be used in the 
li y> 50 illustrated embodiment, with low power processors such as 

a thin single layer polymeric surface loosely wrapped me signelics 87c752 or 87c751, Motorola 68HC11 or 

around a internal hard shell (the hard shell being 68582, or ARM 710 being preferred. If convenient, copro- 

constructed from polymeric or metallic materials). For cessors such as analog to digital converters or digital signal 

example, a nylon or cotton weave, single layer processors can be used alone or in conjunction with a main 
polyethylene, synthetic rubber (with little or no foam 55 processor Conventional flash, static, or dynamic RAM can 

cells present), or natural polymeric materials such as used m the present invention, although for certain applica- 

leather wrapped around a polystyrene casing can be jj ons higher cost embedded DRAM may also be used. In 

used; some storage intensive applications, memory 26 can include 

a composite layered surface having a durable polymeric additional harddisk storage, either located within the device 
outer layer supported by an inner foam layer; or even go 10 or available through an external connection. As will be 

a polymeric bilayer having an intermediate fluid or gel appreciated, for many applications use of optional external 
layer of a viscous or thixotropic material that can be communications can at least partially supplant use of inter- 
used to support extreme deformations. The intermedi- nal processors and memory (except for that necessary to 
ate layer can be relatively thick (one the order of support required sensor or communication buffering and 
centimeters), or in certain embodiments can have a 65 signalling). 

thickness measured on micron to millimeter scales. The present invention optionally supports communica- 

Such extremely thin layers would allow complex tions with an external computer system 40 using its internal 
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communications system 32 and associated transceiver 34. 
The external computer system 40 also includes a transceiver 
42, a personal computer or workstation 44, and is connected 
to a local or wide area network computer system 46. The 
transceivers 34 and 42 can support various communication 5 
protocols and designs, including use of a serial tethered line 

36 (using, for example the RS-232C interface protocols), use 
of infrared signals 38 adhering to widely utilized IRDA 
communication standards, or use of radio frequency signal 

37 (which can be, for example, a cellular telephone, 900 10 
MHz radio, or digital PCS telephonic com munica lions). 
Alternative communication standards, or even alternative 
communication carriers such as those based on optical or 
acoustic techniques, can of course be employed. 

As will be appreciated, in addition to direct communica- 15 
tion with external computer system 40, the device 10 can be 
directly or indirectly maintained in continuous or intermit- 
tent communication with a number of suitably equipped 
electronic devices, including a tablet computer 110, or even 
a physically manipulatable portable computer 11 similar in 
design and function to device 10. Communication can be 
direct to a target device, or through an intermediary retrans- 
mitter such as computer system 40. Other possible commu- 
nication targets include automation control systems, security 
authorization units, personal digital assktants, notebook 
computers, or any other suitably equipped electronic system. 

Results of communications with external devices, presen- 
tation of device stored information, or device status updates 
can all be provided to a user through processor 24 controlled 
update of feedback module 33. Feedback to a user can be 
primarily visual, such as can occur in conjunction with 
visual display 30. Generally, the display 30 can be a con- 
ventional passive or-active matrix liquid crystal display, 
although use of more sophisticated (and expensive) displays 
based on various electrooptical or microraechanical tech- 
niques can of course be used. In addition, for certain devices 
a non-imaging display such as may be formed by a small 
number of status lights (e.g. red or green LEDs), or localized 
or distributed chromatic changes (in conjunction with a 
deformable surface 22 constructed with suitable electrochro- 
mic materials) may be all that is necessary for visual 
feedback to the user. 

In some embodiments of the invention, visual output 
through display 30 may be augmented (or even replaced) 
with a non-visual display 31. The non-visual display 31 can 
include tactile displays based on internal actuators, auditory 
feedback, or even displays based on conformal changes in 
device appearance. For example, one possible feedback 
display is based on internal auditory speakers (emitting a 
range of sounds from simple "beeps" to well formed speech, 
depending on available processor speed and functionality) 
for providing user feedback. As will be appreciated, non- 
visual display 31 and its associated actuators or electronics 
can support alternative feedback modes, including, for 
example, force feedback to a user through internal actuators, 
tactile based feedback (e.g with multiple surface projections 
for presentation of Braille or other conventional tactile user 
interface), modifications to the surface texture of the device, 
or any other conventional mechanism for supplying infor- 
mation to a user. 

To better appreciate operation of the present inventum, 
some selected modes of physical manipulation of device 10 
are schematically presented in FIG. 1. As seen in FIG. 1, 
device 10 can be translationally moved in three spatial 
dimensions, as illustrated with reference to orthogonal force 
arrows 50, 51, and 52. In addition to transl at tonal motion, 
the device 10 can be rotationally moved in any or all three 
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spatial dimensions, as represented by arrows 53, 54, and 55. 
By use of sensor system 28 (alone or in combination with 
communication system 32), the relative or absolute position 
and orientation in three dimensions can be determined. 

In addition to determining spatial position and orientation 
through use of sensors 28, the device 10 can optionally use 
sensor mesh 22 to measure and localize transient or con- 
tinuing force application, with force vectors and related 
timing information being determined and interpreted. A few 
possible force actions (deformation modes) are schemati- 
cally illustrated in FIG. 1, with arrows 60 and 61 illustrating 
depression of surface 20 (with the combination representing 
a squeeze), arrows 62 and 63 illustrating sliding or rubbing 
deformation (with the combination representing a twist), and 
the combination of sliding arrows 65 and 66, and outward 
pull 67, together representing a pinch and outward pull. The 
strength of the applied force can be measured (e.g. a hard or 
soft squeeze is differentiated), its spatial expanse found (e.g. 
to differentiate between poking with a fingertip or a thumb 
pad), and timing determined (e.g. to differentiate between a 
quick or a slow depression of the surface). The deformation 
so caused may be either permanent or transitory. 

As will be appreciated by those skilled in the art, each of 
the foregoing force actions represented by arrows can be 
considered a senseme. Some temporally distinguishable 
sensemes (or combinations of sensemes such as the forego- 
ing discussed pinch/pull combination) further represent mor- 
phemes used as a basis for a morphemic grammar in 
accordance with the present invention. All of the following 
described morphemes can be modified by numerous varia- 
tions in applied pressure, force utilized, appendages used, 
body parts, or extraneous intermediary objects used to apply 
force. In addition, timing of various objects (whether quick, 
slow, or alternately quick and slow) can modify interprela- 
35 tion of a morpheme. For example, if "squeeze" is taken as 
a typical morpheme, one can appreciate various squeeze 
operations such as quick squeeze, slow squeeze, hard 
squeeze, soft squeeze, narrow squeeze, wide squeeze, 
squeeze between hands, squeeze between one hand and a 
40 user's chest or head, squeeze between one hand and a table 
or wall, a squeeze made between two pens or two books, or 
even a squeeze between a user's tongue and the roof of the 
mouth. For purposes of the present invention, all such 
squeeze morphemes would be considered members of the 
45 "squeeze" class, with individual variations acting as possible 
modifiers or selected cases, just as a "house cat" might be 
considered a particular member of the class of "felines", 
which more generally embraces lions, tigers, and bobcats. 
To aid in understanding the diversity of contemplated 
50 physical manipulation morphemes, FIG. 2 illustrates 
selected morphemes arranged by increasing plasticity of the 
device required to enable morpheme utilization, and by 
increasing complexity of available senseme tuples required 
to form or interpret a morpheme applied to a particular class 
55 of device. Beginning with the least plastic device and the 
simplest senseme set used to compose a morpheme, a 
definition of a possible physical manipulation and typical 
function invoked by that manipulation of a device similar 
(but of course possibly more complex) to that described in 
60 conjunction with FIG. 1 is presented: 
Depress 

Definition: Indenting one or more subregions of the 
device through the application of pressure. 

Example: As seen in FIG. 3, consider a device 122 having 
65 a display 123. The device 122 supports a graphical illustra- 
tion software application which allows users to position 
geometric shapes or user defined graphical objects. The 
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device can have four pads 124 placed around it, one on each 
side. By deforming a particular side, the user indicates a 
desire to "nudge" the currently selected geometric shape 125 
away from that side to a new position 126. 
Squeeze 

Definition: To deform one or more subregions of a device 
through the application of vectors of force, some compo- 
nents of which are directed towards each other, which 
compress the morphological structure of the device. This is 
a special case of depression. 

Example: As illustrated in FIG. 4, consider a device 132 
which can display one or more documents in either iconic 
(closed) or textual display (open) modes. By selecting an 
open document 135, and then squeezing a deformable edge 
134 of the device 132, the user indicates a desire to make the 
document "smaller", in this case to iconize it as icon 136. 
Fold 

Definition: To deform a second sub region by bending the 
first subregion such that it partially or completely overlaps 
the second subregion. Further deformations can be applied 
to that new morphological structure on other subregions. 

Example: As illustrated in FIG. 5, consider a device 142 
which can display documents. Suppose this device 142 has 
been augmented such that a deformable horizontal "flap" 
144 on a top edge of the device 142 can be folded to partially 
obscure a display 143. When the user makes this folding 
gesture, the user indicates a desire to password-protect 
("hide") the currently displayed document. 
Curl 

Definition: Deforming one or more subregions of the 
device by spirally configuring subregioas relative to each 
other in a cylindrical or circular morphological structure. 

Example: As illustrated in FIG. 6, consider a device 150 
which can display documents in various languages (English, 
French, etc.). When the user takes such a device 150, curls 35 
in a direction indicated by arrow 157 into a tube, and then 
uncurls it, this "abra-cadabra" gesture tells the device 150 to 
display the current document in a different language. 
Stretch 

Definition: Deforming one or more subregions of a device 40 
through the application of vectors of forces, some compo- 
nents of which are directed away from each other, the 
vectors being applied at opposite ends of the device. 
Example: As illustrated in FIG. 7, consider a device 160 with 
a graphical software application which allows users to 
manipulate geometric shapes. By stretching the device 160, 
the user indicates a desire to "resize" or "rescale" the 
currently displayed shape 165 to a larger size 166, the 
amount of resizing being a function of the amount of 
deformation. Note that SQUEEZING can indicate resizing 
the currently displayed to a smaller size. 
Pinch 

Definition: Manipulating one or more subregions by 
applying vectors of forces, aligned directly towards each 
other, on opposite sides of the affected subregion(s). This is 
typically, but not exclusively, accomplished using two- 
finger tactile force. PINCH is a special case of SQUEEZE. 

Example: As illustrated in FIG. 8, consider a device 170 
which can copy documents. By performing a "pinching 1 ' 
action 175, the user indicates that they wish the next set of 
copies to be issued in stapled form. 
Dogear 

Definition: Deforming a second subregion by folding a 
first subregion at a logical corner or edge of the second 
subregion, indicating a marker or location point (e.g. a 
bookmark) to be later referenced. DOGEAR is a special case 
of FOLD. 



45 



50 



Example: As illustrated in FIG. 9, consider a device 180 
which displays a subset of the pages from a multi-page 
document. By "dogearing" the upper right corner 185 of 
device 180, the user indicates that they wish a bookmark 
associated with the currently displayed page or pages. 
Twist 

Definition: Deforming one or more subregions of the 
device through the application of two opposing rotational 
forces offset from each other by a no n -zero difference about 
some central axis. 

Example: As illustrated in FIG. 10, consider a device 190 
which, over lime, becomes degraded in some aspect of its 
performance (its disk becomes fragmented, its memory 
needs garbage-collection, etc.). By performing a "TWIST" 
gesture 195, the user indicates that they wish the device to 
"wring itself out", performing, for example, garbage collec- 
tion. 

Relief-Map 

Definition: Deforming one or more subregions of the 
device by raising and or lowering them by either spatial 
transformation or the addition/removal of material. 

Example: As illustrated in FIG. 11, consider a device 200 
which can display documents in either one- or two- page 
format. When the user "scores" the device by making a 
vertical indentation about the center axis of the device while 
it displays a single page 206, the device 200 interprets the 
morpheme to request display of documents in a two-page 
format as pages 207 and 208. 
Rip 

Definition: Deforming one or more subregions of the 
device by introducing a spatial discontinuity, by applying 
vectors of force to partially or totally disconnect these 
subregions from the device. 

Example: As illustrated in FIG. 12, consider a device 210 
which can copy some or all of its information. When the user 
performs the "ripping" gesture 215, removing one or more 
subregions, the device 210 copies its currently selected data 
set onto those subregions 216 and 217. 
Perforate 

Definition: Deforming one or more subregions of the 
device by means of introducing a change in the spatial 
connectivity of the subregions such that a hole is introduced 
(either temporarily or permanently) in the device. 

Example: As illustrated in FIG. 13, consider a device 220 
which is used to route messages between various parts of a 
device network, and which displays this functionality to the 
user by means of lines 222 representing message pathways. 
When the user perforates the device 220 with a finger or 
object 224 to introduce a hole in one of these pathways 222, 
the system stops routing messages along that pathway. 
Similarity 

Definition: Deformation of one or more subregions of a 
device previously configured to represent some other pre- 
defined object. Typically the device acts in a manner con- 
sistent with the behavior of the real-world object when it is 
manipulated in this manner. 

Example: As illustrated in FIG. 14, consider a device 230 
which contains text-io-speech and audio input capability; 
and which presents itself lo the user in an anatomically 
accurate shape of a human head. When the user opens lips 
232 on the human head, internal sensors detect opening of 
the lips and activate text-to-speech capability. 
3D Map 

Definition: The morphing of a device that can be molded 
around an external object to permit a sensor mesh to 
determine simultaneously size and shape of the external 
object. The range of possible external objects is large but is 
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limited by the size of the device's solid inner-housing and 
the volume of the moldable material in the outer housing. In 
this system the device has the ability to accurately sense the 
amount of material from its inner surface to the outer edge 
of the moldable material (example: via ultrasonic sounding, 
similar to sonar in water) thus determining an accurate 
electronic model for the shape of the molded enclosure. 

Example: As illustrated in FIG. 15, by pressing a device 
attached moldable material 244 around the surface of an 
external object (e.g. a cog 242) a device 240 can automati- 
cally generate a CAD model of that object and store it in its 
memory. 
Mimicry 

Definition: Deforming one or more subregions of the 
device such that the resultant morphological structure 
resembles a known real -world object and through this asso- 
ciation of subregions, the device acts in a manner consistent 
with the object it resembles. 

Example: As illustrated in FIG. 16, consider a device 250 
containing a computer which contains text-to-speech and 
audio input capability, and which presents itself to the user 
as a misshapen blob having the consistency and plasticity of 
moldable putty or clay. When the user performs the "MIM- 
ICRY" action by molding part of the device 250 to resemble 
an ear, the audio input capability is activated. 

In addition to morphemes based on physical 
manipulation, various morphemes based on varying degrees 
of relative or absolute spatial positioning are contemplated 
to be useful in practice of the present invention. To aid in 
understanding the diversity of contemplated spatial 
morphemes, FIG. 17 illustrates selected spatial morphemes 
arranged by increasing knowledge of spatial position 
required to enable morpheme utilization, and by increasing 
complexity of available senseme tuples required to form or 
interpret a morpheme applied to a particular class of device. 
Beginning with a device having only rudimentary relative 
positioning functionality for supporting simple spatial 
sensemes, and ending with a device absolutely positionable 
to within centimeters anywhere on Earth, a definition of a 
possible spatial manipulation and typical function invoked 
by that manipulation of a device similar (but of course 
possibly more complex) to that described in conjunction 
with FIG. 1 is presented: 
Translate (Relative to Device) 

Definition: The linear movement of a device's center of 
mass from one position in space to another. 

Example: Used to substitute for mouse controlled graphi- 
cal "sliders" in conventional graphical user interfaces. When 
it is only physically possible to display a small amount of 
list, large listings can still be searched by "scrolling'* a 
display window in response to the TRANSLATE mor- 
pheme. 
Shake 

Definition: Spatially translating all subregions of a device 
by repeated movement in opposing directions, such that the 
net translation is negligible. 

Example: As illustrated in FIG. 18, consider a device 260 
which is used as a calculating device. When the user 
performs the "SHAKE" gesture, the device 260 clears its 
accumulator. 
Revolve 

Definition: Rotating all subregions of a device by rotating 
the subregions about a point internal to the device, about any 
arbitrary plane. 

Example: As illustrated in FIG. 19, consider a device 270 
which displays an imaged slice of volumetric data, such as 
medical data from a series of CAT scans. By rotating the 
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device about a center point 272 internal to the device 270 to 
a new position 274, the plane specifying the imaged slice is 
changed accordingly. 
Tilt 

5 Definition: Rotating one or more subregions of a device 
by rotating the subregions such thai one or more components 
of the rotary force are in the direction of gravity and the 
amount of rotation is between about -180 degrees and +180 
degrees. 

io Example: Consider a device which displays frames from 
an animation sequence on the side facing the user. As the 
device is tilted away, the speed of the animation increases — 
as its tilted towards the user, the speed of the animation 
decreases, analogous to operation of a gas pedal. 

15 Flick 

Definition: A forwards TILT immediately followed by an 
opposing backwards TILT. 

Example: As illustrated in FIG. 20, consider a device 280 
which can transmit some subset of its data to another device. 

20 When the user performs the "FLICK" gesture by quickly 
tilting in the direction of arrow 282, followed by a reverse 
tilt along arrow 282, the device 280 performs this 
transmission, towards the device (not shown) pointed to by 
the ray of the gesture. 

25 Spin 

Definition: Rotating one or more subregions of the device 
by rotating the subregions about a point internal to the 
device, such that the plane of rotation is one of the device's 
surface planes. SPIN can be considered a special case of 

30 REVOLVE. 

Example: As illustrated in FIG. 21, consider a device 290 
which can display a frame of video 295 from a video 
sequence. The user performs the "SPIN" gesture in a 
counter-clockwise direction, the device displays earlier 

35 frames in the sequence; when the gesture is performed in a 
clockwise direction (arrow 292), the device 290 displays a 
later frame 296 in the sequence (represented by film strip 
294). 
Orient 

40 Definition: Rotating one or more subregions of the device 
by rotating the subregions about the center of the device, 
such that the plane of rotation is one of the device's surface 
planes, and the amount of the rotation is a multiple of 90 
degrees (i.e. to rotate the device between cardinal compass 

45 points). Orient can be considered a special case of SPIN, 
which in turn is a special case of REVOLVE. 

Example: As illustrated in FIG. 22, consider a device 300 
which can display a document in either 1 page, 2-page, or 
4-page formal ("1-up", "2-up", or * ; 4-up"). When the user 

5U performs the orient gesture in a clockwise direction (arrow 
302), the device 300 increases the number of document 
pages it is displaying from one page 305 to two pages 306 
and 307. Further orient gestures would increase the number 
of displayed pages. When performed in a counter-clockwise 

55 direction, the device 300 decreases the number of pages it is 
displaying. 
Face 

Definition: Manipulating one or more subregions of the 
device such that a first set of subregions is no longer 

60 bottom-most and a second distinct set of subregions now 
assumes the first subregions former position. 

Example: As illustrated in FIG. 23, consider a device 310 
which displays documents, and which allows users to edit 
such documents. Consider further the case in which the 

65 device presents itself to the user in the form of a cube, in 
which 6 different documents are displayed on the 6 different 
faces. When the user performs the "FACE" gesture by 
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making a particular face top-most, the document which is some point exterior to the physical boundaries of the device, 

now on the top-most face becomes editable by the user, where said point is proximal to a body feature of the user, 

while the document which is no longer top-most is no longer This is a special case of ORBIT. 

editable Example: As illustrated in FIG. 26, consider a device 340 

y ft 5 which can perform audio output. By performing the ORBIT 

Definition: The movement of a device's center of mass in ' g^re (hi ' direction indicated by arrow 344) about the 

adirectionoppositetothecuxrentgravitauonforceactingon u f r ' s ear 345 the audio output is activated. 

, j . Movement in Room 

tne device. _ Definition: The local detection of a device's 3D position 

Example: Commanding the device to display the com- tQ reference points found within an enclosing room, 

puters file system at a position one level higher in the 10 Differences m the mea sured position are used to trigger 

hierarchy. actions. 

Pan Example: A virtual filing system that allows you to save 

Definition: The application of a TRANSLATION to a and files based on the devices current position in the 

device such that it is moved parallel to the front of the user's room j 0 save a fij c ycm m ight think carefully about the 

body, at a substantially constant height. 15 contents of the file and then walk to the position in the room 

Example: Viewing a spread-sheet on a device with a that might be most easily associated with it. When retrieving 

display so small only one cell can be shown. By PANing the the file you would use the same thought processes and go 

device, the contents of the current row can be shown in back to the position you had associated with the file. On 

sequence depending on the rate or amount of PAN. doing so the files associated with that position would be 

However, if the device were rotated away from its current 20 displayed and you would then be able to select the file you 

orientation during the PAN, a new row would be chosen. The were looking for. The system Ls useful because the human 

selection of the row could be dependent on the deviation mind is very good at remembering information that is 

from the original orientation. s P aliall Y organized rather than in some abstract information 

Push Pull c ' ala struclure - 

Definition: Manipulating one or more subregions of the 25 Movement Between Widely Separated Sites 

deviee by spaUally translating them such that they are Definition: Mampulating one or subregions of the dev ce 

, . r i- r -7- c <u , , r such that the detected absolute spatial position of those 

moved along a line of projection from the center ol the ^ ^ Ls ^ 

device to the vertical axis of the user s body. Example: Consider a device which can display informa- 

Example: Consider a device with audio output capability. d(m fmm a database of client in f orma tion. When the device 

As the device is "pushed" further away from the body, Us 30 ^ mQVed {Q a different clicnt sitc> tnc devicc automatically 

audio output level increases. When it's "pulled" towards the upc iates its display to display information for the nearest 

body, its level decreases. client site. 

Whack In addition to morphemes based on physical manipulation 

Definition: The application of an accelerative or or spatial positioning, various morphemes based on sensed 

de-accelerative force to one or more subregions of a device 35 environmental conditions are contemplated to be useful in 

such that said subregions contact or are contacted by an practice of the present invention. To aid in understanding the 

external object, causing an equal and opposite countering diversity of contemplated environmental morphemes, FIG. 

force. 27 illustrates selected environmental morphemes loosely 

Example: As illustrated in FIG. 24, consider a device 320 arranged in order of increasing sensor complexity needed in 

which can perform long and unpredictable database 40 some commonly sensed environmental categories. For each 

searches. When the user performs the WHACK gesture (e.g. of the proffered categories, some selected sensing systems 

upon a table 322), the current search is aborted. supportable by a device such as that described in conjunc- 

Orient (Relative to Environment) lion with FIG. 1 are presented: 

Definition: Manipulating two subregions of the device Light 

such that the line drawn between the centers of those two 45 Definition: Manipulating one or more subregions of the 

subregions alters its orientation with respect to the surround- device such that the amount of light falling upon ihose 

ing environment. subregions changes. 

Example: Displaying a CAD drawing of a machine -part in Example: Consider a device used in a lecture hall to take 

3D on the display of a mobile device. As the orientation of notes. When the room lights are turned on, the light sensors 

the device changes, so too does the viewing angle and 50 detect this and turn down the backlight to conserve energy, 

position of the rendered image. When the room lights are turned off (e.g. during a slide 

O r bit show) the light sensors detect this and turn up the backlight 

Definition: Rotating one or more subregions by rotating to increase viewabilily. 

the subregions and/or the center of mass of the device about light sensors can range from thresholded binary light 
some point exterior to the physical boundaries of the device, 55 detectors, to light pattern detectors, to full imaging systems, 

about any arbitrary axis. Advanced techniques can include image analysis and rec- 

Example: As illustrated in FIG. 25, consider a device 330 ognition to identify objects or persons, 

which can search a network database, such as the World Heat 

Wide Web, for information. When the user performs the Definition: Manipulating one or more subregions of the 
"orbit" gesture, such a search is initiated. The radius 332 of 60 device such that the amount of heat applied to those subre- 

the rotation 334 specifies the breadth of the search— wider gions changes. 

circles specify a wider search. The speed of the gesture Example: Consider a portable computer which has a 

specifies the time limit imposed on the search— the quicker stylus for entering text. By looking at the heat profile along 

the gesture, the more cursory the search. . the back surface of the computer, the computer can detect 
Orbit Relative to User 65 whether it is being held with the left hand, the right hand, 

Definition: Rotating one or more subregions by rotating both hands, or neither hand, and update its interface accord- 

the subregions and/or the center of mass of the device about ingly. 
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Thermal (heat) sensors can range from simple tempera- of the same basic database. When the user MATCHes the 

ture sensors to sophisticated differential thermal mappers first device 360 to the second device 361 followed by 

and thermal imagers. matching the third device 362 to the matched first and 

Electromagnetic second devices, their databases are reconciled 

Definition: Manipulating one or more subregions of the 5 (synchronized), 

device such that the electromagnetic spectrum applied to stack 

those subregions changes. t Definition: To move one or more subregions of a device 

Example: By analyzing the radio spectrum, the device can such mat tncy enter physical contact with a subregion of a 

derive estimates as to its absolute spatial position, and use seC ond device, such that the first device is now located 

that to alter its functionality. 10 above, but physically adjacent to, the second device. Or, to 

Electromagnetic detection can include magnetic uke tWQ ^ aligned and remove (i.e. unstack) that 

compasses, radio detection, or GPS signal detection. More alienmcnt 

advanced techniques can include electromagnetic spectrum Example- As illustrated in FIG. 31, consider a set of 

analysis and interpretation, such as roughly determining deviccs 370( 371( an d 372, each of which is displaying a 
location based on available radio signals. ^ framc of y £ co f ^ om fl bngcr video sequcn ce. When the 

Vibrate devices are stacked, the ordering of the stacking specifies an 

Definition: Manipulating one or more subregions ol the * >™ { ^ ^ si le composil6 vide0 is 

device by vibration. . . 

Example: Consider a device which displays textual infor- now produced, 

mation Wheo the user lakes the device on the bus, the Tile \, 
Ambient vibration level sensed by the device changes, and 20 Definition: To move one or more subregion? of a dev ce 

fhedevice increases the si Z e of the displayed textto help the such that they physically contact with >*e^» 

ate second device, such that the first device and second device 

" This class of environmental morphemes can include now form a single seamless spatial unit or to lake two 

detection of intermittent contacts, low frequency rumblings, devices so aligned and remove lhat alignment. TILE is a 
or acoustic level detection. More advanced techniques 25 special case of MATCH. 

requWnTaeater processor power include maximum fre- Example: As illustrated in FIG. 32 consider a set of 

oTncy identification, spectral analysis of acoustic frequen- devices 380, 381, 382, 383, each of which «nmdq»£»tiy 

ties (enabling the device to distinguish background envi- display a portion of a arge photograph. When J^m» 

ronmental noises from speech, for example), or even speech are tiled, each device displays he portion of the Photograph 
bTd identification of persons in the vicinity of a device. 30 appropriate to its current relative position in the tiled grid. 

In addition to morphemes based on physical Relatively Align 

manipulation, spatial position, or sensed environmental Definition: To move one or more subregions of device 

?M various morphemes based on cooperation between such that they engage ma particular spatial relation to . one 

Sjle interacting devices are contemplated to be useful in or more other devices where said devices ^« ~^*»»« • 

practice of the present invention. To aid in understanding the 35 Example: As illustrated in FIG. 33, consider a set of 

d vS of contemplated spatial morphemes, FIG. 28 illus- devices 390, 391, 392, and 393 which are displaying a 

rrMerlSvice morphemes arranged by an increasing multi-page document, 

level of possible physical contact, and by increasing com- at the far left (device 390) displays the tab e of conlenU 

plexity of available senseme tuples required to form or whichever one is presently placed at the far nghi device 

interpret a morpheme applied to a particular class of device. 40 393) displays the index, and the others display pages accord- 

B Snin£ w° h a device having only rudimentary edge ing to their respective locations. As d.flerent devices can 

deformation Snc.iona.ity for supporting simple spatial have different display capabilities, mov.ng them about can 

sensemes, and ending with complex delormabk or alter the document display. For example, .1 only one ol the 

mbeddable devices thai can be wrapped about each other. devices has a color display when ,, » -novedtrom «*ond 

^definition of a possible multidevice manipulation and 45 position to ih.rd position then (a) the device when was n 

typical function invoked by that multidevice manipulation of third position, displaying page #2, now displays i page #1^ 

devices similar (but of course possibly more complex) to and (b) the color display, which was displaying page #1, now 

that described in conjunction with FIG. 1 is presented: displays page #2 in color. 

Touc h Wrapping/Embedding 

Definition: To move one or more subregions of a device so Definition: Manipulating one or more subregions of a 

such that they enter physical contact with a subregion of a device such lhat these subregions spatially occlude or are 

s^ond device," any alignment and to any extent. Or, to spatially occluded by some portion of the second device. 

Se two devices so Signed and remove that alignment. Example: As Ulustrated in FIG. 34 consider a first device 

Ixmple: Consider two portable computers 350 and 351, 400 which contains the infrastructure to support the filtering 

the firs, of which contains" a database, and the second of 55 of email. Consider a second set of devices 401 to and 402 

which contains an IRDA port. When the user touches the which implement particular email filters. When the user 

S^mputer 350 to the second computer 351, the database physically embeds device 401 (or 402) into the first device 

Transmuted via the second computer's port. 400, by which the first device 400 now wraps the se«md 

Match device 401 (° r m )' P articular emai1 fllter su PP orted b V 

Definition: To move one or more subregions of a device eo the second device is activated, 

such tba. they enter physical contact with one or more As those skilled in the art will appreciate, combinations of 

ubreg o s of a second'device, with the subregion(s) of the any of .he foregoing described morphemes base km , ptay. - 

first device and the subregion(s) of .he second device being cal manipulation, spatial position, environmental condit ions, 

ahgned along one or mort edges. Or. to take two devices so or multiple interacting devnees can be extended by pamc- 

5ed and remove thai alignment. 65 palion in a morphemic -sentence". A sentence ,s denned as 

Example: As Ulustrated in FIG. 30, consider, multiple a sequence of one or more temporally disjoint morphemes, 

devices 360 361, and 362, which contain differen. versions Typically, be.ween about '/,olh ol a second and 2-3 seconds 
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suffices to distinguish morphemes within a sentence. Of Database Presentation Sentence 

course, in some circumstances and indefinite time period Consider a device which contains various personal infor- 

may elapse. The sentence level allows definition of a physi- rnatioo databases, such as a list of phone numbers, a list of 

cal manipulatory grammar by appropriate choice of a mor- addresses, and a calendar. Suppose that the user wishes the 

pheme sequence, and corollary rules governing, for 5 mos t appropriate of those databases displayed. Then a 

example, use of active (verb like) morphemes, naming gestural sequence to support this could be: 

(noun) morphemes, or connectors. Just as the position and SPATIAL LOCATION — the user carries the device such 

relation of words in a sentence define the sentence's mean- that it ^ spat ially proximate to either the telephone, the 

ing (e.g. "horse chestnut" is not the same as "chestnut address book, or the refrigerator (where the famQy calendar 

horse"), similarly the position and relation of morphemes m 1Q fa dis laycd)> whichever is appropriate. 

a manipulatory sentence define the sentence s meaning. For DEPRESS — the user touches the device to activate the 

SSSfe m ' ™ mmum ? t Uon f mod H f h w,l °P^tion. The device now displays the personal information 

WHACK could mean "transfer data and erase the local * • * . .u . ■ 

copy", while a WHACK followed by a FLICK means appropriate to that _ loca ion. 

"power on the device and transfer data". In other settings, a Da f abasc R f " cval Sentence , - , , k 

FLICK or a WHACK could mean something entirely dif- 15 To cxtcnd the foregoing example ot a database presenta- 

ferent. To better understand construction of morphemic lion sentence, consider two computers which contain calcn- 

sentences, the following examples are discussed: dar databases. If the user wishes to synchronize the 

Data Transfer Sentence calendars, a suitable gestural sequence to support this could 

Consider a device which can transmit some or all of its be: 

information to another device. Furthermore, this transmis- 20 SQUEEZE — the user squeezes the device to activates its 

sion can be done either unencrypted, or encrypted (to gestural recognition capabilities. 

increase security). Furthermore, this transmission of a text+ ORBIT — the user orbits the device about the surface of the 

graphics document can either include the graphics, or omit other device 3 times, indicating a desire to only match data 

them (to save time). Suppose that the user wishes to perform f or the next 3 weeks. 

the command "Transmit the information in document A, 25 MATCH — the user matches the edge of the device to the 

encrypted, omitting graphics, to machine B". Then a gestural e dge of the calendar, indicating a desire to "match" contents 

sequence (morphemic sentence) to support this could be: between the two devices. 

DEPRESS — the user presses on a displayed representa- Printer/Copier Control Sentence 

tion of A, indicating that A is to be selected for an upcoming Consider a device which can produce paper copies of 

operation 30 documents. Suppose that the user wishes to tell such a 

FLICK — the user flicks the device in the direction of device to produce a stapled, two sided copy of document A, 

device B, indicating that the operation is a transmission to B enlarged to the next greater size. Then a gestural sequence 

FOLD — the user folds the top quarter of the device over to support this could be: 

the lower three-quarters, indicating that the transmission is DEPRESS — the user presses on a displayed representa - 

to be encrypted. 35 tion of A, indicating that A is to be selected for an upcoming 

TWIST — the user twists the device about its central axis, operation 

indicating that the data is to be "wrung out", i.e. the graphics rip — the user introduces a spatial discontinuity into a 

are to be omitted portion of the device, indicating that the upcoming operation 

SQUEEZE — the user squeezes the device, indicating that is to be a copy ("carrying away" some of the data), 

they are confirming that the operation is to proceed. 40 PINCH — the user pinches the upper left corner of the 

Note that none of these gestures, in isolation, performs an device, indicating that the copies are to be stapled, 

act — the "ensemble" of temporally separated morphemes SQUEEZE — the user presses on the front and back of the 

must be interpreted in order to form the complete action. device, indicating that the copy is to be two-sided. 

Drawing Modification Sentence STRETCH — the user stretches the device, indicating thai 

Consider a device which displays geometric shapes tor 45 the copy is to be an enlargement to the next greater size, 

user manipulation. Furthermore, one of the manipulations FACE — the device typically has its paper emitter on the 

supported is to resize (or rescale) a shape. Furthermore, bottom, preventing users from making copies accidentally, 

suppose that this resizing can be done either aliased (jaggy) By facing the device such that the emitter is on the side, the 

or anti-aliased (edges are smoothed). Suppose that the user copying operation is initiated, 

wishes to perform the command "Resize shape A by 120%, 5u Light Based Control Sentence 

about the X axis only, using anti-aliasing". Then a morphe- Consider a device which can display documents. Suppose 

mic sentence to support this could be: the user is using the document while seated on a train, and 

DEPRESS — the user presses on a displayed representa- wishes the document to display itself with a backlight when 

tion of A, indicating that A is to be selected for an upcoming the train enters a tunnel, and wishes the document to display 

operation 55 itself in a larger font when the train rumbles over rough 

STRETCH — the user stretches some portion of the tracks. Then a gestural sequence to support this could be 

device, indicating that the operation is to be a resize. When SQUEEZE — the user squeezes on the device, indicating 

the user starts stretching, a portion of the status display that a loss of light is to be compensated for by a backlight, 

displays "100". The user continues to stretch until the status LIGHT — as the train enters a tunnel, the LIGHT gesture 

display reads "120" 60 is made, and the device turns on the backlight. 

RELIEF-MAP — the user "scores" the display by making WHACK — the user briskly raps the device against the 

a horizontal line of depression, indicating that the palm of their hand, indicating that their desired preference 

operation is to take place only about the horizontal (X) for adjusting hard-to-read documents is to increase the font 

axis. size. 

DEPRESS — a circular thumb stroke is made in a different 65 VIBRATE — as the train goes over a bridge, the vibratory 

area of the device, indicating that anti-aliasing gesture is sensed. Because of the position of this VIBRATE 

(smoothing the edges) is to be performed. morpheme (after the preceding WHACK to gesture) in this 
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morphemic sentence, the device now increases the font size 
on the displayed text. 

LIGHT — the user puts the device into his suitcase, mak- 
ing the LIGHT gesture. In this context (with no SQUEEZE 
before it), the LIGHT gesture causes the device to power-off 5 
its display. 

To better appreciate utility and construction of devices in 
accordance with the present invention, several examples of 
devices are now described: 

Portable Computer with Squeeze and Tilt Control 3Q 

A handheld portable computer 500 (e.g. a 3Com ® 
PalmPilot ®) capable of being fitted with deformable, pres- 
sure sensitive edging 504 is schematically illustrated in 
FIGS. 35 and 36. The computer 500 supports a name and 
address software application, providing a user viewable 
name-address entry field on display 503. In this 15 
embodiment, a user can squeeze the deformable, pressure 
sensitive edging 504 (squeeze arrows 507) of the computer 
500. In response, the name and address software application 
causes the display 503 to animate by slowly incrementing 
(scrolling) through the name list from "A" towards "Z" 20 
entries. When the user squeezes edging 504 again, the 
software application stops the scrolling animation. Scrolling 
functionality is further enhanced by the use of a tilt sensor, 
which allows the computer's behavior to mimic conven- 
tional rotatable address books. If the computer 500 is tilted 2 s 
away from the 45 degree angle at which someone might 
typically hold it, the scrolling rate is increased. In this 
application, the closer the computer 500 was tilted towards 
the user (as indicated by arrow 506 of FIG. 36), the faster the 
scroll rate toward "Z". However, if a user tilted the computer 
500 back past the neutral 45 degree position (as indicated by 
arrow 506 in FIG. 36), the animation would move back- 
wards with a velocity related to the magnitude of lilt. In this 
way it was possible for a user to search for items in a long 
list in a very natural way, while only using one hand. 

In an aftemative mode schematically illustrated in FIG. 35 
37, scrolling speed can be completely controlled by pres- 
sure. The greater the squeeze pressure (arrows 537), the 
faster the list scrolls. Release of the applied pressure causes 
the scrolling to halt. In this alternative user interface 
strategy, application tilt (as indicated by orthogonal tilt 40 
arrows 530 and 532) could be used to change the direction 
of the scrolling through the list, allowing a user to search 
portions of a large two dimensional data set (schematically 
illustrated as a data plane 520) that is not entirely visible 
either horizontally or vertically on display 503. By simply 45 
tilting the display 503 of computer 500 as if it were a 
window through which the data plane 520 can be viewed, 
any particular portion of the data plane (for example, data 
subset 524) can be viewed. As will appreciated, in both the 
foregoing modes the speed of scrolling, the specific neutral 50 
tilt angle, and required pressures to initiate scrolling changes 
can be adjusted to fit a particular user. 
Portable Computer with Handedness Detection 

Pressure sensors have been added 10 augment a conven- 
tional keyboard 551 enabled user interface to a hand hold- 55 
able Windows ® CE class computer 550 (i.e. a Cassio ® 
Cassiopia ®) schematically illustrated in FIGS. 38 and 39. 
In this embodiment, user handedness was determined by 
using pressure sensors positioned on a right back-edge and 
a left back-edge of computer 550. User studies have found so 
that the difference in pressure between the right and left side 
gave a direct indication of handedness of a user. As respec- 
tively illustrated in FIGS. 38 and 39, handedness was used 
to justify formatted text 554 to the left (FIG. 38) or right 
(FIG. 39), thereby allowing more space 555 on display 553 65 
for an electronic annotation pen to be used to mark-up the 
text. 
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For both the embodiment of the invention illustrated by 
FIGS. 35-37, and the foregoing embodiment illustrated in 
FIGS. 38 and 39, material deformation of a spongy, elastic, 
or otherwise deformable material must be measured. 
Although various lechniques can be used to measure mate- 
rial deformation, including those based on imaging or fluid 
volumetric changes, one particularly useful Icchnique is 
based on the use of pressure transducers. Commercially 
available sensors measure pressure (indicative of material 
deformation) by converting a pressure change to a change in 
electrical characteristics. For example, inexpensive sensors 
that change resistance in response to pressure can be 
obtained in a variety of shapes and sizes, including paper 
thin sensors, and easily bend able sensor strips. Sensors of 
this kind can be customized to any particular shape or form 
that a gestural Ul might require. The change in resistance is 
usually linearly related to pressure, with the sensor generally 
being placed in a potential divider network to model the 
pressure as a change in potential. For a practical circuit the 
resulting signal needs to be amplified, buffered and trans- 
lated such that the change in value from minimum pressure 
to maximum pressure spans a useful range. The modified 
signal can now be fed into an analog to digital converter 
(ADC) to produce a digital representation of pressure. An 
8-bit ADC can typically be used for most applications, 
however, if greater sensitivity to pressure changes is 
required, a higher resolution ADC (e.g 16-bit ADC) can be 
used. As will be appreciated, the ADC could be memory 
mapped into the processor's address space as a peripheral, or 
alternatively supplied as a retrofitted pressure interface to 
existing computers capable of benefiting from this system. 
Since an RS232 connection is an interface that is almost 
universally available on portable computers, one strategy is 
to have the parallel output of the ADC converted into a serial 
RS232 frame using a serializer such as a UART and then 
level-shift and buffer the signal as specified by the RS232 
standard. At the computer end of the serial interface, another 
level-shifter and UART, the output of which is readable by 
the processor, performs the serial-to-parallel conversion. 

As described with reference to FIG. 40, in realizing a 
working system a microcontroller 564 (a Signetics 87c752) 
with an ADC built in to combine many of the I/O tasks in a 
single chip can be used in combination with level shifter 566 
(a MAX3223). This approach has the advantage that intel- 
ligent processing of the input signal is possible with soft- 
ware. Tilt measurement was provided by a tilt sensor 567 
connected to a buffer/amp 562 to supply an analog signal to 
microcontroller 564. The pressure measurement can also be 
encoded within a protocol across the serial link. This par- 
ticular microcontroller 564 has five ADC inputs, but by 
making use of eight digital control lines, it is possible to use 
only one ADC input and one buffer amplifier 561, to 
measure up to eight pressure points with pressure sensors 
565. This is achieved by using the control lines to select only 
one sensor at a time and take a reading for each using a 
single input to the ADC. After eight sensors have been 
selected, eight readings are acquired in memory. The design 
is practical because the microcontroller can take measure- 
ments and make analog to digital conversions at a rate far 
higher than is needed to communicate with a host computer 
569. 

For scrolling or handedness based software applications, 
16 levels were determined to be adequate for representing a 
pressure measurement. In order to have a high data through- 
put to the host computer 569, each measurement was 
encoded in one byte of an RS232 frame such that the four 
lowest bits were the pressure representation and the highest 
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four bits were the sensor ID. Each frame of the RS232 data can contain a video segment in independent memory. Shuf- 
was therefore completely self contained. Of course any fling or reorganizing the tiles can allow users to physically 
protocol that limits itself to a specific number of devices in manipulate the sequence of video segments to affect a 
its address space will eventually have the problem that physically manipulatable video editing system. Using the 
someday applications might be designed that need to refer- 5 carc i analogy, tiles can be used to re-sequence documents, 
ence far more devices than are supported. The solution used pages within a document, audio annotations, voice mail, or 
in this protocol is to reserve the sensor ID number IS as a otncr temporal media contained within the tiles. The result- 
special value that can extend the semantics of the encoding ant scquencc can then be played as a whole by using the tiled 
to include an arbitrary number of bytes representing a sensor structure as a unit. 

or value. For the described software applications, the com- 1Q For purposcs 0 f the present invention, display tile array 
monly used RS232 frame format (1 start, 8 data no parity, 1 configurations can be categorized as follows: 
stop bit at a baud rate of 9600) was chosen. Close-packed display tile array 600 (FIG. 43) in which 
In operation, the host computer 569 determined handed- dlcs 6Q2 arc arranged to span a surface 610 in closely 
ness by utilizing information about the current pressure abutting, but not overlapping relationship, such that the 
exerted on two subregions— one, on the back of the device, 15 ^^0^ display area is maximized (i.e. seamlessly). As 
occupying roughly the left half, and the other, also on the win be apprcc iatcd, the tiles could themselves form a 
back of the device, occupying the right half. The current f rces tanding continuous surface, or they could be disposed 
pressure values were converted from analog to digital form, on a ublc of othcr su i ta ble support. Each of the tiles 602 
such that a digital value of 0 (zero) represents no pressure, SU p por ts a display 604 sized to substantially cover a front 
and a digital value of, for example, 15 represents maximum 2() surface of each t * ilc 60 2. In certain embodiments, a back 
pressure. The detection circuitry then proceeds: surface of each tile 602 can also support a display, 
if (left_sensor is high AND righl_scnsor is high) then Advantageously, this would allow for creation of free stand- 
conclude that the user is gripping the device with both j n g displays with images visible on the front and back. The 
hands surface 610 can be a plane, a sphere, or any arbitrary shape 
else if (left__sensor is high AND right__sensor is low) then 2 5 that permits tiling. 

conclude that the user is gripping the device with the Loose-packed display tile array 620 (each tile 622 having 

left hand only a display 624 as seen in FIG. 44) in which tiles 622 are 

else if (left„sensor is low AND right_sensor is high) then slotted into a lattice pattern (indicated by dotted lines 625). 

conclude the user is gripping the device with the right Each tile can be considered to be situated in a regular 

hand only 30 bounded lattice slot (namely, the lattice slot that contains a 

else if (left_sensor is low AND right_sensor is low) then defined center of each of the tiles) within the lattice where 

conclude that the user is gripping the device with the dimensions of the bounded slot are no more than a few 

neither hand times the maximum dimension of the tile, and there is no 

Also, to optimize communication, pressure values are only more than one tile in any slot. Within the lattice regions any 

sent when they change. In order to account for jitter and 35 of the tiles can be positioned arbitratrily and still retain the 

error in the pressure sensors, a sensor is only considered same inter-relationship with the group. It is still possible for 

'high' if its value is higher than some minimum threshold tiles to touch one another at the boundaries of two or more 

(e.g. '2' on the 0 to 15 range). lattice slots, however, this is not a requirement. 

Scanner/Printer/copier Supporting Edge Deformable Dis- Free-format display tiles 630 (each tile 632 having a 

pl a y 40 display 634 as seen in FIG. 45) are similar to loose-packed 

Ascanner/printer/copier device 570 is schematically illus- display tiles 620 where the size and shape of the lattice slots 

trated in FIG. 41. As seen in FIG. 41 (and in more detail in may vary freely (e.g. the dimension of a lattice slot may be 

FIG. 42) the device 570 supports a display 574 having a many times the maximum dimension of any of the partici- 

deformable edge 572, with the combination roughly shaped pating tiles). The only constraint on the arrangement is that 

like a piece of paper. In operation, a user can place a written 45 there must be no ambiguous relationships about the relative 

document in the device 570 for scanning. An electronic connectivity of the each of the tiles 632. That is to say. a tile 

version of the scanned document is displayed (i.e. as text that is to display the next piece of information to one side of 

575 in FIG. 42) on the display 574. By outwardly pulling the another tile must be unique and not be confused with the task 

deformable edge 572 as indicated by arrow 577, a user can of another tile in the tiling lattice. 

instruct the device 570 to resize the document before print- 50 3D display tiles (packable display tiles) are created by 
ing or copying. Pinching opposing sides of the deformable extending the three foregoing display tile categories. How- 
edge 572 (arrows 578) can further instruct the device 570 to ever for close packed tiling of display tiles, packed devices 
print double sided copies. As will be appreciated, various in the center of a 3D structure would not be available as the 
other morphemes can be used to provide a simple interface user interface. This may not matter as the surface of the 3D 
for interacting with device 570. 55 shape will expose a area that will have unique affordances 
Tileable and Stackable Portable Displays for some classes of application. For instance, cubic tiles 
Multiple autonomous display tiles having an onboard packed into the shape of a large cube can display, using the 
display controller and at least one surface consisting sub- six faces of the larger cube, the various projections that 
stantially of a conventional display are particularly useful could be rendered by viewing a 3D CAD drawing from each 
for the practice of various aspects of the present invention. 60 degree of freedom. 

Such tiles can be interconnected in response to various As will be appreciated, display tiles need not be recti- 

morphemes such as TOUCH, FLICK, RELATIVELY linear but can be hexagonal, circular, or of arbitrary shape 

ALIGN, or WHACK, or even can be operated without and size. The size of tiles need not be constant for all tiles 

substantial user mediated morphemic input in certain situ- within a larger structure. Tiles need not be aligned but may 

auons . 65 require proximity to indicate adjoining points or edges. 

Advantageously, tile positioning can be used as an inter- Tiles need not be in physical cootact to define they are 

face specifier in and of itself. For example, each display tile joined in a group activity but instead this may be a pro- 
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grammed function. The connectivity can also be discovered modulated, or those based on code division multiple access) 
through a radio network from a coordinating server or from and operate at a range of transmitter powers. There is no 
a distributed algorithm that draws in as many computers as longer a need for direct alignment if the system is designed 
necessary for a task, using the wireless network to negotiate with suitable communication tolerances. The transmitter 
the resources it needs. The tiled computers may also be 5 range plays a crucial part in the design. If there is only 
connected by wired networking systems, although in cases enough power in an emitted signal to be picked up within a 
where the topology might need to rapidly change, this is not few millimeters of a tile edge, then the signals will be 
as desirable as a wireless system. An example of a wired isolated, the topology will be defined by the physical con- 
networking system of this type is a system that uses the nectivity and the complexity of designing the system to 
intermit to include many computers in a single task, one in 10 avoid interference from neighboring signal sources will be 
which each computer knows its relative position even minimized. However, an alternative design is to use more 
though they may be separated by rooms or in the extreme powerful radios. In this case all tiles will be able to contact 
cases buildings, cities or countries. However, in general the all other tiles and inter-tile connectivity needs to be defined 
most useful case of display tiling is when the tiles are by another parameter. Signal strength can be used or, more 
proximate enough that they can all be viewed by one person 15 deliberately, information that relates the ID of a tile to a 
to create a display medium whereby the viewing experience spatial map (perhaps held in one master tile) describing the 
is an enhancement over using a single display. Accordingly. position of all tiles in the tile array. In this system it Ls also 
tiles can operate as either a single larger contiguous structure necessary to minimize inter-tile interference. For digital 
or they can retain individual properties and independent packetdata systems that operate at the same frequency, 
functions or a combination of both, (e.g., jumbo-tron like 20 carrier sense multiple access (CSMA-CD or CSMA-CA) 
function, 12x12 small displays as used in a TV studio, or systems are well known techniques to solve this problem, 
picture in-picture features found in commercial TVs or Other solutions involve tiles using different frequencies, 
editing suites). with frequencies reused depending on the power of the 

Depending on the type of packing, various schemes can transmitters. This is the technique used by traditional eel- 
be used to allow for permanent, intermittent, or even one- 25 hilar telephones. Yet another approach is to use Code 
time communication between display tiles. For example, Division Multiple Access (CDMA) that relies on the over- 
close packed tiling can make use of wired connectivity laying of signals in the same region of the EM spectrum, a 
between computers or could use a variety of wireless or technique known as spread spectrum modulation, 
optical communication technologies. In the case of wired For loose packed display tiles, the wireless techniques 
connectivity, edge mounted conventional plug and socket 30 described above in connection with close packed tiling 
connectors may be used to create a rigid tiled array. Plug and generally become essential to the implementation. However, 
socket systems lend themselves to parallel connections for a special case of loose packing exists in which each edge of 
bulk and high-speed data transfer. They also provide a a tile does make contact with every other surrounding tile, 
convenient method of power distribution, which can allow except that it may only be a single point of contact and not 
for one of the tiles providing a power source that supports 35 at a accurately defined place. A wired version of this system 
the rest of the array. A plug and socket connection between can be built in which the entire edge of each tile is a serial 
tiled computers 692 in a tiled computer array 690 (with both connection including one of the two Vertices that define the 
data and power transfer shown) is schematically illustrated edge. Communication in each direction can be achieved by 
by lines 695 in FIG. 48. a variety of commercially available techniques, including 

The exact design and locations of electrical connectors 40 use of a one -wire interface (plus a ground) for bidirection 

depends on the intended use, and subsequently, the shape of communication between tags and a reader. Note that the 

tile components. For applications where large seams ground connection for a tile arrangement can be derived by 

between displays are acceptable, simple rigid connectors sharing a common ground connection through the surface 

could be attached to the center of each edge, providing the tiles are laid out on. For example, the surface could be 

connectivity to all the surrounding tiles. Other applications 45 made of a metal sheet. The system may be further enhanced 

might require a more complex design. For example, appli- by ensuring that the edge contact is made of a magnetic 

cations which require high-quality information display on a material and the vertices have a magnet embedded in the 

seamless array surface (i.e. tiled blueprints), might employ end. Such an arrangement ensures that there will be a good 

spring-loaded contacts on the four edge connectors. The electrical contact between the transmitter and the receiver, 

spring mechanism allows all array connections to be made 50 The receiver can also derive power from an electrically 

below the display surface, while tiles can still be inserted transmitted signal by bridge-rectifying it and storing the 

and removed from the interior of the array. Removal of tiles collected charge in a capacitor for use by its own electronics, 

could be triggered by some host signal which actuates a Thus, power distribution can also be included in a one-wire 

release of the spring and pops the tile out of the array. interface. In this way flexible connectivity can be achieved 

Serial connectivity can also be used in a close packed 55 to support the rapid and convenient rearrangement of tiles in 

arrangement such as that illustrated in FIG. 43. It has the which only the minimum amount of care needs to be taken 

advantage that fewer connections have to be made and in in setting up connectivity. 

practice it might be more reliable. However the net band- Loosely packed tile displays such as illustrated in FIG. 45 

width between tiles will be less than a parallel system. Serial may require that the display surfaces use a best effort 

communication lends itself to optical and wireless systems 60 algorithm to present a unified display with all the sections of 

thus removing the need for any physical connections. For the displayed image bearing the correct spatial arrangement 

optical technologies alignment of the transmitter and to each other even though, in the case of rectangular tiles, 

receiver is still important although the clever use of light they might not be registered vertically or horizontally and 

pipes and lens capture techniques can introduce more Ilex- have an offset angle relative to each other. In order to 

ibility. Wireless systems can use many different bands of the 65 implement the desired tile display algorithm, not only is the 
EM spectrum (kHz, MHz, GHz), utilizing a variety of relative arrangement of tiles important but the exact offset 

modulation techniques (amplitude modulated, frequency (distance and angle) from each other is also important. There 
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are several methods that are suitable for automatically 
determining offsets between loosely packed tiles. For 
example, as illustrated with respect to FIG. 46 (showing 
communicating tiles 652 and 654, with respective displays 
651 and 653), optical encoding 660 along an edge can be 
used to identify tile orientation. It is possible to use a binary 
coded optical pattern along an edge which is regular and 
encodes the distance from a vertex at any point. A tile that 
abuts, or is relatively aligned, can read this pattern using 
optical sensors 658 and 659 and determine the display offset 
in the direction of the edge. Alternatively, as illustrated by 
FIG. 47 (showing communicating tiles 672 and 674), it is 
possible to use radio based techniques that rely on signal 
strength triangulation. Each vertex 675 or 676 of a tile 672 
can contain a wireless transmitter and a receiver. If these 
vertices transmit a short characteristic wireless signal at well 
known times, a nearby tile 674 can use the receivers situated 
at its own vertices 680, 681, 682, and 683 to triangulate the 
position of each transmitting vertex relative to them by 
measuring the relative delay by which the signal was 
received. After two of the vertices of the original transmit- 
ting tile 672 have signaled, the adjoining tile 674 can 
determine its exact position and orientation within their local 
region of a tile array. The transmitting and receiving tiles can 
now swap roles, with the result that both tiles will know their 
relative position. This process can be extended across the tile 
array. 

Free format tiling differs from loose-packed tiling in that 
there are no proximity, or regular format, constraints. To 
demonstrate how a free format tiling system might work, the 
following example is described. Imagine a number of laptop 
computers each fitted with a Global Positioning System 
(GPS) and a radio modem. Each laptop can determine its 
position (with acceptable error) and communicate it to all ^5 
the other laptops by contacting them through the radio 
modem. After some period of time the laptops will all know 
their relative location and absolute location. If ;tny comput- 
ers changes its location, it can update the local neighbors to 
ensure there is enough understanding in the array to consider 40 
the computers are in a known tiling configuration even 
though they are not close to each other and in fact they may 
be in different geographic regions. An application that might 
use this free format tiling system is one that wishes to ensure 
information is being correctly sent in a uniform and dis- 
persed fashion across a very large area. For example, the 
information that each laptop computer receives could be 
instructions to release an amount of a pesticide used to 
control an insect that does damage to commercial crops. If 
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Daisy Chain Routing 

Display tiles are arranged to have a logical connectivity 
with each other so that each one only transmits information 
to the next in a predefined line. The computers are said to be 
daisy chained to each. Any information sent to the start of 
the chain contains an ID and the first computer in the chain 
compares it to its own ID. If it matches, it acts on the data. 
If not, it sends the data on to the next computer in the chain 
until it finds its destination. 
N-ary Routing 

In N-ary routing Ihe path to the destination is contained in 
the ID of the device. A simple routing example is schemati- 
cally illustrated with reference to the direction arrows 695 in 
FIG. 48, which shows quaternary routing in physically 
connected lileable displays. In quaternary routing, an array 
is conceptually arranged as a quaternary tree with each node 
having an input and three outputs. In this system each pair 
of bits of the ID contains a routing command. A 0 indicates 
send the packet to the first output, a 1 indicates the second 
output, a 2 indicates the third output, and 3 indicates no 
further transmission. There is also a count that gets decre- 
mented by each node to tell successive nodes the bit number 
that is currently being considered and when the packet has 
reached its destination. In this way packets are forwarded 
from node to node with a simple choice at each stage until 
they reach their target display. As will be appreciated, it is 
possible to design N-ary systems with more than three 
outputs (a power of two is usually convenient for imple- 
mentation (e.g. 4, 8, 16 . . . )). 
Flooding 

Flooding has no predefined routing structure. The com- 
puter that takes the first packet from the master checks to see 
if it has the correct ID. If not, the packet is sent out on all 
links lo which it has not yet sent or received that packet. The 
result is a Hood of copies of the packet across the array 
eventually reaching its destination. The packets must also 
have a maximum hop count to ensure they eventually are 
removed from the system. The disadvantage of this approach 
is that many more tiles are burdened with the processing of 
unwanted data than in the previous two schemes, which may 
impact the overall efficiency of the system. 
Hot Potato 

The Hot Potato algorithm is similar to the Flooding 
algorithm, except a retransmitted packet is sent out on only 
one output that is either chosen randomly or is the least busy. 
The process stops when the packet reaches the correct tile. 
The time that a packet takes to reach its destination is not 
deterministic. 

As those skilled in the art will appreciate, other various 



the pesticide is released in too high a concentration in any 50 modifications, extensions, and changes to the foregoing 



locality, it may be hazardous to human health. The tiled 
approach allows the computers to roam (e.g. in the back of 
a truck), displaying information as to the type and concen- 
tration of pesticide that is to be released given their current 
relative proximity. 55 

Various conventional algorithms can be used to support 
distribution of information between autonomous tiled dis- 
plays. These algorithms assume a system by which there is 
a master controller generating data lo be displayed. There is 
also a large array of tiled computers that the system will use eo 
to display visual data and/or process information. Each 
computer in the tiled array contains a unique ID. It is the job 
of the master to split the data into pieces that each computer 
can display and for this information to be packaged along 
with the ID of the target computer. The algorithms below 65 
describe bow the information travels from the master to a 
destination display tile: 



disclosed embodiments of the present invention are contem- 
plated to be within the scope and spirit of the invention as 
defined in the following claims. 
The claimed invention is: 

1. A plurality of tileable devices for transferring data, 
comprising 

a first device having a display, a processor, and a first 
communication module for transferring daia; 

a second device having a display, a processor and a 
second communication module for transferring data; 

a third device having a display, a processor and a third 
communication module for transferring data; 

wherein the first device is connected in substantially 
simultaneous communication with the second device 
and the third device to pass data based on spatial 
positions of the respective first device, second device, 
and third device, and 
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the first device maintains information about spatial posi- 
tion of the second device and the third device, and 
transfers differing data to the second device and the 
third device based on that maintained spatial position 
information, wherein the spatial position information 5 
maintained by the first device includes, for each of the 
second and third devices, information defining a rela- 
tive spatial position between that second or third device 
and the first device. 

2. The plurality of tileable devices of claim 1, wherein the 
first, second and third communication modules respectively 
support optical data links. 

3. The plurality of tileable devices of claim 1, wherein the 
first, second and third communication modules respectively 
support radio links. 

4. The plurality of tileable devices of claim 1, wherein the 15 
first, second and third communication modules respectively 
support physical electrical connections. 

5. The plurality of tileable devices of claim 1, each first, 
second and third device respectively further comprising a 
first, second, and third deformable piece, with intercommu- 20 
nication between at least two of the devices being triggered 

by a touching action between devices. 

6. The plurality of tileable devices of claim 1, each first, 
second and third device respectively further comprising a 
first, second, and third deformable piece, with intercommu- 25 
nication between at least two of the devices being triggered 
by a stacking action between devices. 

7. The plurality of tileable devices of claim 1, each first, 
second and third device respectively further comprising a 
first, second, and third deformable piece, with intercommu- 
nication between at least two of the devices being triggered 30 
by a tiling action between devices. 

8. A plurality of tileable devices for transferring data 
comprising 

a first device having a display, a processor and a first 
communication module for transferring data, 

a second device having a display, a processor and a 
second communication module for transferring data, 

a third device having a display, a processor and a third 
communication module for transferring data, ^ 

wherein the first device is maintained in at least intermit- 
tent communication with the second device, with the 
first device maintaining information about spatial posi- 
tion of the second device and the third device, and 
transferring differing data to the second device and the 45 
third device based on that maintained spatial position 
information, and the second device maintaining infor- 
mation about spatial position of the third device, and 
with the first device implementing a first data set, a 
second data set being transferred from the first device 50 
to the second device for implementation, and a third 
data set being transferred from the second device to the 
third device for implementation, therein the spatial 
position information maintained by the first device 
includes, for each of the second and third devices, 5S 
information defining a relative spatial position between 
the second or third device and the first device, and the 
spatial position information maintained by the second 
device including information defining a relative spatial 
position between the third device and the second 
device. 



9. The plurality of tileable devices of claim 7, wherein the 
first, second and third communication modules respectively 
support optical data links. 

10. The plurality of tileable devices of claim 8, wherein 
the first, second and third communication modules respec- 
tively support radio links. 

11. The plurality of tileable devices of claim 8, wherein 
the first, second and third communication modules respec- 
tively support physical electrical connections. 

12. The plurality of tileable devices of claim 8, wherein 
the first, second and third device respectively further com- 
prising a first, second, and third deformable piece, with 
intercommunication between at least two of the devices 
being triggered by a touching action between devices. 

13. The plurality of tileable devices of claim 8, each first, 
second and third device respectively further comprising a 
first, second, and third deformable piece, with intercommu- 
nication between at least two of the devices being triggered 
by a stacking action between devices. 

14. The plurality of tileable devices of claim 8, each first, 
second and third device respectively further comprising a 
first, second, and third deformable piece, with intercommu- 
nication between at least two of the devices being triggered 
by a tiling action between devices. 

15. A dynamic array of devices, the array comprising, 

a plurality of devices, each device having a display 
maintainable in a user determined relationship to each 
other, and each device having a determinable informa- 
tion state, and 

a plurality of position detectors, each device having at 
least one of the plurality of position detectors usable 10 
determine its position relative to one or more of the 
plurality of devices, wherein in response to each device 
determining its relative position, that device transmits 
the determined relative position to at least one other 
device, 

wherein determinable information state of each of the 
plurality of devices is modified upon its change of 
position relative to one or more devices. 

16. A tileable display system, comprising 

a first device having a display, a processor, and a first 
communication module for transferring data, 

a second device having a display, a processor and a 
second communication module for transferring data, 
wherein: 

the first device is connected in substantially simulta- 
neous communication with the second device to pass 
data based on spatial positions of the respective first 
device and second device, and 

the first device maintains information about spatial 
position of the second device, and selectively trans- 
fers data to the second device based on the main- 
tained spatial position information, wherein the spa- 
tial position information includes information 
defining a relative spatial position between the first 
and second devices. 
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